Vehicle Classification¶
About the Notebook¶
I have approached this case study the following way
- Started with the data visualization, EDA and then worked on the Data Imputation with the help of sklearn Imputer
- In modelling I first build SVM with the original data (All 18 Independent variables).
- Then in dimensionality reduction used PCA to reduce the Independent variables
- Build SVM classifier based on PCA reduced dimension and compared the accuracy, precision, recall, F1 scores with the classifier built on the original data
About data:¶
The data has 18 independent variables (all continuous variables) and 1 Target variable which has 3 values namely Car, Bus and Van
Steps and Tasks:¶
- Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm
- Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why
- Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn)
- Train a Support vector machine using the train set and get the accuracy on the test set
- Perform K-fold cross validation and get the cross validation score of the model
- Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data
- Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state)
- Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings
In [1]:
#Import all the necessary modules
import pandas as pd
import numpy as np
import os
from sklearn.cluster import KMeans
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from sklearn import svm
from sklearn.decomposition import PCA
from sklearn.metrics import classification_report, confusion_matrix
from sklearn import metrics
from scipy.stats import zscore
import plotly.express as px
import plotly.graph_objects as go
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from plotly.subplots import make_subplots
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
In [2]:
# Read the vehicle data
veh = pd.read_csv("C:/Users/tt002/Downloads/vehicle.csv")
veh.head()
Out[2]:
In [3]:
veh.info() # gives infomarion on data type and total non null data in the column
In [4]:
veh.describe() # shows the descriptive statistics of the data set
Out[4]:
Insights¶
- 4 variables namely distance_circularity, max.length_aspect_ratio skewness_about1 and Scaled_variance has very high Standard deviation compared to other variables. This show the spread of these variables are high and very diversified
1. Data Preprocessing¶
In [5]:
veh.isna().sum() # Check for missing values in each column
Out[5]:
In [6]:
# fill missing values with median column values
X = veh.drop('class',axis=1) # Predictor feature columns (18 X m)
Y = veh['class'] # Predicted class (van, bus, car) (1 X m)
imp = Imputer(missing_values='NaN', strategy='median', axis=0) # override default behavior to replace Nan with Median
imp_x = pd.DataFrame(imp.fit_transform(X),columns = list(X))
In [7]:
imp_x.isna().sum() # Check if all the missing values are imputed
Out[7]:
2. Understanding the attributes¶
In [8]:
# shape of the independent variables
fig, ax = plt.subplots(3, 6, sharex='col', sharey='row',figsize = (20,10))
m=0
for i in range(3):
for j in range(6):
imp_x.hist(column = imp_x.columns[m],ax=ax[i,j])
m+=1
Insights¶
- 3 independent variables have double peak (bimodal) namely Distance circularity, elongatedness and Scatter ratio, meaning there could be overlapping of classes in these dimensions
- 3 independent variables are right skewed namely Scaled variance 1 & 2, scaled radius of gyration_1, meaning there are extreme values on higher side which could possible be outlier which can be confirmed with a Box plot
In [9]:
# Pair plot with all the variables
plt.figure(figsize=(50,50))
sns.pairplot(veh)
Out[9]:
Insights¶
- There is a very high multicolinearity in the dataset except of 6 - 7 variables(out of 18) namely (skewness_about, about1,2, hollow ratio, pr axis, maximum length aspect) all other variables are one or the other point correlated with any of the other variables
3. Spliting the dataset into training and test set¶
In [10]:
x_train, x_test, y_train, y_test = train_test_split(imp_x, Y, test_size=0.3, random_state = 2)# 3. spliting the data into training and test set
4.Training the SVM on the training data set¶
In [11]:
# fitting SVM model on training data
clf = svm.SVC(gamma=0.025, C=3)
mod = clf.fit(x_train, y_train)
In [12]:
y_pred = clf.predict(x_test) # predicting SVM model on test data
print("The model accuracy in training set:{0:.4f}".format(mod.score(x_train, y_train)))
print("Model Accuracy in Test set: {0:.4f}".format(metrics.accuracy_score(y_test, y_pred)))
Insights¶
- This is a classical case of high variance or overfitting. The model has 100% accuracy in training set but performs as equivalent to a random guess in testing set. In this the model the SVM classifier used the default Radial base kernel, lets try out with polynomial kernel and see how it works
In [13]:
clf_poly = svm.SVC(gamma=0.025, C=3, kernel = 'poly') # changed the default rbf to polynomial kernal
mod = clf_poly.fit(x_train, y_train)
In [14]:
y_pred = clf_poly.predict(x_test)
print("The model accuracy in training set:{0:.4f}".format(mod.score(x_train, y_train)))
print("Model Accuracy in Test set: {0:.4f}".format(metrics.accuracy_score(y_test, y_pred)))
Insights¶
- The change kernal for classification has dramatically improved the accuracy from a near random accuracy of 51% to around 97% which is excellent
5. Perform Kfold crossvalidation¶
In [15]:
# Kfold Cross Validation
kfold = KFold(n_splits= 10, random_state=2)
model = svm.SVC(gamma=0.025, C=3, kernel = 'poly')
results = cross_val_score(model, imp_x, Y, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
Insights¶
- The Average cross validation accuracy of 10 test sets are around 96% with a standar deviation of 2.5%. Hence we can certain that our model will perform at minimum 93% + accuracy when deployed in production
6. Dimensional reduction with Principal Component Analysis to caputre 95% variation¶
In [16]:
plt.figure(figsize=(25,8))
sns.boxplot(data = imp_x,orient = 'h') # Box plot to check how different variable are spread across
Out[16]:
Insights¶
- Variables are of different magnitude hence need to be normalized before performing a PCa to nullify any effect of single hig magnitude variable
In [17]:
df =imp_x.apply(zscore) # replacing the original values with z scores to normalize the data
df.head()
Out[17]:
In [18]:
plt.figure(figsize=(25,8))
sns.boxplot(data = df,orient = 'h') # To Check how the data is normalized post transformation
Out[18]:
Insights¶
- All the variables are normalized now with same level of magnitude
In [19]:
# PCA
# Step 1 - Create covariance matrix
cov_matrix = np.cov(df.T)
print('Covariance Matrix \n%s', cov_matrix)
Identify Eigen values¶
In [20]:
# Step 2- Get eigen values and eigen vector
eig_vals, eig_vecs = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s', eig_vecs)
print('\n Eigen Values \n%s', eig_vals)
In [21]:
tot = sum(eig_vals)
var_exp = [( i /tot ) * 100 for i in sorted(eig_vals, reverse=True)] # get the % of variation explained by each principal component
cum_var_exp = np.cumsum(var_exp) # cummulative summation of variation
print("Cumulative Variance Explained", cum_var_exp)
In [22]:
plt.plot(var_exp)
plt.title('SCREE PLOT')
plt.xlabel('Principal Component number')
plt.ylabel('Eigen value')
Out[22]:
Insights¶
- Visually we can observe that their is steep drop in variance explained with increase in number of PC's. the elbow point comes around 3rd principal component. Since our goal is to choose principal components that together explain over 95% variance, lets plot the cummulative variance plot
In [23]:
# Ploting
plt.figure(figsize=(10 , 5))
plt.bar(range(1, eig_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eig_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Insights¶
- Upto 7th / 8th Principal component explain over 95% of the variation, lets take a conservative approach and go with top 8 principal components for building our SVM model
In [24]:
#we are generating only 7 PCA dimensions (dimensionality reduction from 18 to 8)
pca8 = PCA(n_components=8)
pca8.fit(df)
print(pca8.components_)
print(pca8.explained_variance_ratio_)
pca8 = pca8.transform(df)
In [25]:
df_pca = pd.DataFrame(pca8) # Principal components - top 8 components
df_pca.head()
Out[25]:
In [26]:
sns.pairplot(df_pca)
Out[26]:
Insights¶
- No correlation can be seen between any of the variables unlike the multicollinearity seen between the original variables
7.Building SVM model using 8 Principal components¶
In [27]:
# splitting the principal component data into test and training set
x_train_pca, x_test_pca, y_train_pca, y_test_pca = train_test_split(pca8, Y, test_size=0.3, random_state = 2)
clf_pca = svm.SVC(gamma=0.028, C=3, kernel = 'poly') # using Polynomial kernal as it gave an excellent accuracy with original data
mod_pca = clf_pca.fit(x_train_pca, y_train_pca)
print("The model accuracy in training set:{0:.4f}".format(mod_pca.score(x_train_pca, y_train_pca)))
In [28]:
x_train_pca, x_test_pca, y_train_pca, y_test_pca = train_test_split(df_pca, Y, test_size=0.3, random_state = 2)
clf_pca = svm.SVC(gamma=0.028, C=3, kernel = 'rbf')
mod_pca = clf_pca.fit(x_train_pca, y_train_pca)
print("The model accuracy in training set:{0:.4f}".format(mod_pca.score(x_train_pca, y_train_pca)))
In [29]:
# Kfold Cross Validation
kfold = KFold(n_splits= 10, random_state=2)
model = svm.SVC(gamma=0.025, C=3, kernel = 'rbf')
results = cross_val_score(model, df_pca,Y, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
8. Insights on Model accuracy.¶
- The Principal components are spread entirely different from the original variables because of which the polynomial kernal that worked well in the original variables scored pathetically with the PCA data. Whereas the Radial Kernal which has a accuracy near equal to random accuracy is having a higher accuracy in models with top 8 principal components.
- The Average cross validation accuracy of 10 test sets are around 94.5% for 8 Principal components as against a 96% accuracy with 18 original independent variables
- The standar deviation of 2.5% is same as that of the standard deviation got through cross validation of moedl with 18 original independent variables. This conludes the model accuracy is not significantly affected by reducing the original variables to 8 principal components
Confusion matrix, precision, accuracy, f1 score¶
In [30]:
y_pred = clf_poly.predict(x_test)
print("The model accuracy in training set:{0:.4f}".format(mod.score(x_train, y_train)))
print("Model Accuracy in Test set: {0:.4f}".format(metrics.accuracy_score(y_test, y_pred)))
print(classification_report(y_test, y_pred))
cmsvm=metrics.confusion_matrix(y_test, y_pred, labels=["bus","car","van"])
df_svm = pd.DataFrame(cmsvm, index = [i for i in ["bus","car","van"]],
columns = [i for i in ["Predict bus","Predict car","Predict van"]])
ax1=sns.heatmap(df_svm, cmap="Accent", annot=True,fmt = '')
ax1.title.set_text('Support Vector Machine - Polynomial Kernel')
In [31]:
y_pred_pca = clf_pca.predict(x_test_pca)
print("Model Accuracy in Test set: {0:.4f}".format(metrics.accuracy_score(y_test_pca, y_pred_pca)))
print(classification_report(y_test_pca, y_pred_pca))
cmsvm_pca=metrics.confusion_matrix(y_test_pca, y_pred_pca, labels=["bus","car","van"])
df_svm_pca = pd.DataFrame(cmsvm_pca, index = [i for i in ["bus","car","van"]],
columns = [i for i in ["Predict bus","Predict car","Predict van"]])
ax1=sns.heatmap(df_svm_pca, cmap="Accent", annot=True,fmt = '')
ax1.title.set_text('Support Vector Machine - PCA - 8 components')
Conclusion¶
- We build model with both original data set with 18 independent variables and reduced dataset with 8 principal components and compared the below metrics
- 10 Fold cross validation average accuracy
- 10 Fold cross validation Standard deviation of accuracy
- Precision
- Recall
- F1 Score
- Confusion Matrix
- We couldn't see any significant drop in any of the accuracy metrics. The difference is just between 1 - 1.5% in all the metrics mentiond above. Hence the PCA had done well in reducting the dimension by retaining the important elements without affecting the classification accuracy